This Notebook has been adapted from an training authored by Esri, found here and offered under Apache License (2.0).
Original Copywrite Notice:
Licensing¶
Copyright 2018-2022 Esri
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
A copy of the license is available in the repository's license.txt file.
Significant changes made by NCAR:
collection and create_movie functions to accept an output directoryThe World is changing daily. Major changes like shrinking of lakes, river path shifts, construction of megastructures can be seen directly from the satellite images. This notebook creates a movie to visualize monthly changes in Hirakund reservoir, Odisha.
First, let's import all the necessary libraries and connect to our GIS via an existing profile, e.g. gis = GIS("Pro")
# Find the directory of the current project and add to PATH
import sys, os, arcpy
home_folder = arcpy.mp.ArcGISProject("current").homeFolder
sys.path.insert(0, home_folder)
os.chdir(home_folder)
# The 00_environment_setup notebook contains libraries and other things common to all the notebooks (e.g. file paths)
%run "00_environment_setup.ipynb"
# Additional libraries
from platform import system
from arcgis.raster.functions import apply
# Because we will later import Image from PIL, we need to import IPython.display.Image as something else
from IPython.display import Image as dImage
Active Portal in ArcGIS Pro Logged in as ksampson Current conda environment: arcgispro-py3-clone C:\Users\ksampson\AppData\Local\ESRI\conda\envs\arcgispro-py3-clone Found input data directory: C:\Users\ksampson\Documents\GitHub\GloFAS_Q2Q_Bias_Correction_and_Verification\data\input Completed importing and/or installing libraries in 4.65 seconds.
Note: to run this sample, you need a few extra libraries in your conda environment. If you don't have the libraries, install them by running the following commands from cmd.exe or your shell
conda install -c anaconda pillow
conda install -c conda-forge imageio
try:
from PIL import Image, ImageFont, ImageDraw
except ImportError:
print('Could not import pillow library. Installing using conda.')
#!conda install -c anaconda pillow -y
!conda install --freeze-installed esri::numpy=1.20.* anaconda::pillow -y
from PIL import Image, ImageFont, ImageDraw
try:
import imageio
except ImportError:
print('Could not import imageio library. Installing using conda.')
#!conda install -c conda-forge imageio -y
!conda install --freeze-installed esri::numpy=1.20.* conda-forge::imageio -y
import imageio
Could not import imageio library. Installing using conda.
Collecting package metadata (current_repodata.json): ...working... done
Solving environment: ...working... done
## Package Plan ##
environment location: C:\Users\ksampson\AppData\Local\ESRI\conda\envs\arcgispro-py3-clone
added / updated specs:
- imageio
The following packages will be downloaded:
package | build
---------------------------|-----------------
blas-1.0 | mkl 1 KB conda-forge
cffi-1.16.0 | py39ha55989b_0 231 KB conda-forge
cryptography-41.0.4 | py39hb6bd5e6_0 1.0 MB conda-forge
freetype-2.12.1 | hdaf720e_2 498 KB conda-forge
imageio-2.31.5 | pyh8c1a49c_0 284 KB conda-forge
libsqlite-3.43.2 | hcfcfb64_0 827 KB conda-forge
m2w64-gcc-libgfortran-5.3.0| 6 342 KB conda-forge
m2w64-gcc-libs-5.3.0 | 7 520 KB conda-forge
m2w64-gcc-libs-core-5.3.0 | 7 214 KB conda-forge
m2w64-gmp-6.1.0 | 2 726 KB conda-forge
m2w64-libwinpthread-git-5.0.0.4634.697f757| 2 31 KB conda-forge
msys2-conda-epoch-20160418 | 1 3 KB conda-forge
openssl-3.1.3 | hcfcfb64_0 7.1 MB conda-forge
pip-23.3 | pyhd8ed1ab_0 1.3 MB conda-forge
setuptools-68.2.2 | pyhd8ed1ab_0 454 KB conda-forge
sqlite-3.43.2 | hcfcfb64_0 830 KB conda-forge
------------------------------------------------------------
Total: 14.3 MB
The following NEW packages will be INSTALLED:
ca-certificates conda-forge/win-64::ca-certificates-2023.7.22-h56e8100_0
imageio conda-forge/noarch::imageio-2.31.5-pyh8c1a49c_0
libblas conda-forge/win-64::libblas-3.9.0-1_h8933c1f_netlib
libcblas conda-forge/win-64::libcblas-3.9.0-5_hd5c7e75_netlib
libiconv conda-forge/win-64::libiconv-1.17-h8ffe710_0
liblapack conda-forge/win-64::liblapack-3.9.0-5_hd5c7e75_netlib
libsqlite conda-forge/win-64::libsqlite-3.43.2-hcfcfb64_0
libzlib conda-forge/win-64::libzlib-1.2.13-hcfcfb64_5
m2w64-gcc-libgfor~ conda-forge/win-64::m2w64-gcc-libgfortran-5.3.0-6
m2w64-gcc-libs conda-forge/win-64::m2w64-gcc-libs-5.3.0-7
m2w64-gcc-libs-co~ conda-forge/win-64::m2w64-gcc-libs-core-5.3.0-7
m2w64-gmp conda-forge/win-64::m2w64-gmp-6.1.0-2
m2w64-libwinpthre~ conda-forge/win-64::m2w64-libwinpthread-git-5.0.0.4634.697f757-2
msys2-conda-epoch conda-forge/win-64::msys2-conda-epoch-20160418-1
ucrt conda-forge/win-64::ucrt-10.0.22621.0-h57928b3_0
vc14_runtime conda-forge/win-64::vc14_runtime-14.36.32532-hdcecf7f_17
The following packages will be UPDATED:
cffi pkgs/main::cffi-1.15.1-py39h2bbff1b_3 --> conda-forge::cffi-1.16.0-py39ha55989b_0
cryptography esri::cryptography-41.0.3-py39_2 --> conda-forge::cryptography-41.0.4-py39hb6bd5e6_0
jpeg esri::jpeg-9e-0 --> conda-forge::jpeg-9e-hcfcfb64_3
libdeflate 1.8-h2bbff1b_5 --> 1.17-h2bbff1b_1
libpng pkgs/main::libpng-1.6.37-h2a8f88b_0 --> conda-forge::libpng-1.6.39-h19919ed_0
libtiff 4.5.0-2 --> 4.5.1-0
lz4-c pkgs/main::lz4-c-1.9.3-h2bbff1b_1 --> conda-forge::lz4-c-1.9.4-hcfcfb64_0
numpy esri::numpy-1.20.1-py39_0 --> conda-forge::numpy-1.20.3-py39h6635163_0
numpy-base 1.20.1-py39_0 --> 1.24.3-py39_1
olefile pkgs/main::olefile-0.46-pyhd3eb1b0_0 --> conda-forge::olefile-0.46-pyh9f0ad1d_1
openssl esri::openssl-3.0.10-0 --> conda-forge::openssl-3.1.3-hcfcfb64_0
pillow 9.5.0-py39_0 --> 9.5.0-py39_1
pyopenssl pkgs/main/win-64::pyopenssl-23.2.0-py~ --> conda-forge/noarch::pyopenssl-23.2.0-pyhd8ed1ab_1
setuptools esri/win-64::setuptools-67.7.2-py39_0 --> conda-forge/noarch::setuptools-68.2.2-pyhd8ed1ab_0
sqlite esri::sqlite-3.41.2-0 --> conda-forge::sqlite-3.43.2-hcfcfb64_0
vc pkgs/main::vc-14.2-h21ff451_1 --> conda-forge::vc-14.3-h64f974e_17
vs2015_runtime esri::vs2015_runtime-14.27.29016-h5e5~ --> conda-forge::vs2015_runtime-14.36.32532-h05e6639_17
wincertstore pkgs/main/win-64::wincertstore-0.2-py~ --> conda-forge/noarch::wincertstore-0.2-pyhd8ed1ab_1009
zlib esri::zlib-1.2.13-0 --> conda-forge::zlib-1.2.13-hcfcfb64_5
zlib-ng esri::zlib-ng-2.0.6-1 --> conda-forge::zlib-ng-2.0.7-hcfcfb64_0
zstd pkgs/main::zstd-1.5.2-h19a0ad4_0 --> conda-forge::zstd-1.5.5-h12be248_0
The following packages will be SUPERSEDED by a higher-priority channel:
blas pkgs/main --> conda-forge
certifi pkgs/main/win-64::certifi-2023.7.22-p~ --> conda-forge/noarch::certifi-2023.7.22-pyhd8ed1ab_0
freetype esri::freetype-2.12.1-4 --> conda-forge::freetype-2.12.1-hdaf720e_2
lerc esri/noarch::lerc-4.0-pyh39e3cac_0 --> conda-forge/win-64::lerc-4.0.0-h63175ca_0
libxml2 esri::libxml2-2.10.4-arcgispro_0 --> conda-forge::libxml2-2.10.4-hc3477c8_0
pip pkgs/main/win-64::pip-23.3-py39haa955~ --> conda-forge/noarch::pip-23.3-pyhd8ed1ab_0
pycparser pkgs/main::pycparser-2.21-pyhd3eb1b0_0 --> conda-forge::pycparser-2.21-pyhd8ed1ab_0
six esri::six-1.16.0-py_0 --> conda-forge::six-1.16.0-pyh6c4a22f_0
tzdata pkgs/main::tzdata-2023c-h04d1e81_0 --> conda-forge::tzdata-2023c-h71feb2d_0
wheel pkgs/main/win-64::wheel-0.41.2-py39ha~ --> conda-forge/noarch::wheel-0.41.2-pyhd8ed1ab_0
Downloading and Extracting Packages
blas-1.0 | 1 KB | | 0%
blas-1.0 | 1 KB | ########## | 100%
blas-1.0 | 1 KB | ########## | 100%
imageio-2.31.5 | 284 KB | | 0%
imageio-2.31.5 | 284 KB | ########## | 100%
imageio-2.31.5 | 284 KB | ########## | 100%
pip-23.3 | 1.3 MB | | 0%
pip-23.3 | 1.3 MB | 1 | 1%
pip-23.3 | 1.3 MB | ########## | 100%
pip-23.3 | 1.3 MB | ########## | 100%
m2w64-gmp-6.1.0 | 726 KB | | 0%
m2w64-gmp-6.1.0 | 726 KB | ########## | 100%
m2w64-gmp-6.1.0 | 726 KB | ########## | 100%
openssl-3.1.3 | 7.1 MB | | 0%
openssl-3.1.3 | 7.1 MB | ###8 | 39%
openssl-3.1.3 | 7.1 MB | ########## | 100%
openssl-3.1.3 | 7.1 MB | ########## | 100%
m2w64-gcc-libs-core- | 214 KB | | 0%
m2w64-gcc-libs-core- | 214 KB | ########## | 100%
m2w64-gcc-libs-core- | 214 KB | ########## | 100%
cffi-1.16.0 | 231 KB | | 0%
cffi-1.16.0 | 231 KB | ########## | 100%
cffi-1.16.0 | 231 KB | ########## | 100%
cryptography-41.0.4 | 1.0 MB | | 0%
cryptography-41.0.4 | 1.0 MB | ###9 | 40%
cryptography-41.0.4 | 1.0 MB | ########## | 100%
cryptography-41.0.4 | 1.0 MB | ########## | 100%
m2w64-gcc-libs-5.3.0 | 520 KB | | 0%
m2w64-gcc-libs-5.3.0 | 520 KB | ########## | 100%
m2w64-gcc-libs-5.3.0 | 520 KB | ########## | 100%
m2w64-libwinpthread- | 31 KB | | 0%
m2w64-libwinpthread- | 31 KB | ########## | 100%
m2w64-libwinpthread- | 31 KB | ########## | 100%
setuptools-68.2.2 | 454 KB | | 0%
setuptools-68.2.2 | 454 KB | ########## | 100%
setuptools-68.2.2 | 454 KB | ########## | 100%
msys2-conda-epoch-20 | 3 KB | | 0%
msys2-conda-epoch-20 | 3 KB | ########## | 100%
msys2-conda-epoch-20 | 3 KB | ########## | 100%
sqlite-3.43.2 | 830 KB | | 0%
sqlite-3.43.2 | 830 KB | ########## | 100%
sqlite-3.43.2 | 830 KB | ########## | 100%
freetype-2.12.1 | 498 KB | | 0%
freetype-2.12.1 | 498 KB | 3 | 3%
freetype-2.12.1 | 498 KB | ########## | 100%
freetype-2.12.1 | 498 KB | ########## | 100%
libsqlite-3.43.2 | 827 KB | | 0%
libsqlite-3.43.2 | 827 KB | ########## | 100%
libsqlite-3.43.2 | 827 KB | ########## | 100%
m2w64-gcc-libgfortra | 342 KB | | 0%
m2w64-gcc-libgfortra | 342 KB | ########## | 100%
m2w64-gcc-libgfortra | 342 KB | ########## | 100%
Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done
Retrieving notices: ...working... done
Search for Multispectral Landsat layer in ArcGIS Online.
landsat_item = gis.content.get('d9b466d6a9e647ce8d1dd5fe12eb434b')
landsat = landsat_item.layers[0]
landsat_item
Applying Natural color to the filtered Landsat collection using predefined apply function
# India Reservoirs in Living Atlas
CWC_Reservoirs = "34b71f5ea24b49ce857e8ee5e71a4117"
item = gis.content.get(CWC_Reservoirs)
item
%%time
dam_layer = item.layers[0]
dam_layer_filter = dam_layer.query(where="name = 'K.R.Sagara Dam'", as_df=True)
extent = dam_layer_filter.iloc[0]['SHAPE'].extent
extent = {'xmin': extent[0], 'ymin': extent[1], 'xmax': extent[2], 'ymax': extent[3]}
print(extent)
m = gis.map('Mysuru, India')
m.add_layer(dam_layer_filter, options={'opacity':0.1})
m.basemap = 'satellite'
m
{'xmin': 8505585.607700001, 'ymin': 1390822.4349999987, 'xmax': 8524908.163600001, 'ymax': 1408422.3772}
Wall time: 2.18 s
MapView(layout=Layout(height='400px', width='100%'))
rgb_collection = apply(landsat, 'Natural Color with DRA')
The function below creates an array of images with the desired time intervals. If a user specifies 'm' then the images in the selected collection will be consolidated on a monthly basis i.e. all the images of the specified extent will be mosaicked monthly and if the user specifies 'y' as the interval then the images in the selected collection will be consolidated on yearly basis.
from functools import lru_cache
@lru_cache(maxsize=50)
def load_font():
try:
if system()=='Windows':
return ImageFont.truetype("arial.ttf", 30)
elif system()=='Linux':
return ImageFont.truetype("~/.fonts/truetype/dejavu/DejaVuSans.ttf", 30)
else:
return ImageFont.truetype("Arial.ttf", 30)
except:
return ImageFont.load_default()
def collection(df, out_dir, interval, start, end, height, width):
images=[]
if(interval=='m'): # monthly
for i in range(int(start.split('-')[0]), int(end.split('-')[0])+1):
for j in range(1,13):
selected = df[(df['AcquisitionDate'].dt.year == i) & (df['AcquisitionDate'].dt.month == j)]
id = selected['OBJECTID'].values.tolist()
if(len(id)>0):
rgb_collection.mosaic_by(method="LockRaster",lock_rasters=id)
img_name = 'img_'+str(i)+"-"+str(j)+".jpg"
img_file = os.path.join(out_dir, img_name)
rgb_collection.export_image(bbox=extent, size=[height,width], f='image',
save_folder=out_dir,
save_file=img_name)
img = Image.open(img_file).convert('RGB')
font = load_font()
draw = ImageDraw.Draw(img)
draw.text((500, 0),'{:>2}-{:<4}'.format(j,i),(255,255,255),font=font)
images.append(img)
os.remove(img_file)
elif(interval=='y'): # yearly
for i in range(int(start.split('-')[0]), int(end.split('-')[0])+1):
selected = df[df['AcquisitionDate'].dt.year == i]
id = selected['OBJECTID'].values.tolist()
if(len(id)>0):
rgb_collection.mosaic_by(method="LockRaster",lock_rasters=id)
img_name = 'img_'+str(i)+".jpg"
img_file = os.path.join(out_dir, img_name)
rgb_collection.export_image(bbox=extent, size=[height,width], f='image',
save_folder=out_dir,
save_file=img_name)
img = Image.open(img_file).convert('RGB')
font = load_font()
draw = ImageDraw.Draw(img)
draw.text((500, 0),'{:>4}'.format(i),(255,255,255),font=font)
images.append(img)
os.remove(img_file)
return images
The function below will generate a movie (gif) from the collection saved from the above step.
def create_movie(target, out_dir, interval, start, end, height, width, extent, duration):
start_date = datetime.strptime(start, '%Y-%m-%d')
end_date = datetime.strptime(end, '%Y-%m-%d')
selected = target.filter_by(where="(Category = 1) AND (CloudCover <=0.5)",
time=[start_date, end_date],
geometry=arcgis.geometry.filters.intersects(extent))
df = selected.query(out_fields="AcquisitionDate, GroupName, CloudCover, DayOfYear",
order_by_fields="AcquisitionDate").sdf
df['OBJECTID'] = df['OBJECTID'].astype(int)
df['AcquisitionDate'] = pd.to_datetime(df['AcquisitionDate'], unit='ms')
frames = collection(df, out_dir, interval, start, end, height, width)
out_gif = os.path.join(out_dir, 'movie_'+interval+'.gif')
imageio.mimsave(out_gif, frames, format='GIF', loop=0, duration=duration)
print("Movie Created")
return out_gif
%%time
# calling create_movie function for the year 2022 on a monthly basis
#out_gif = create_movie(rgb_collection, output_data_dir, 'm' ,'2022-01-01','2022-12-31', 1000, 1000, extent, 1.0)
# Call the create_movie function for 1980-2023 for annual interval
out_gif = create_movie(rgb_collection, output_data_dir, 'y' ,'1980-01-01','2023-10-01', 1000, 1000, extent, 1.0)
Movie Created Wall time: 1min 50s
The movie (gif) will be created in the out_dir directory specified in the create_movie call. The gif below is generated using the code above which shows how the Hirakund reservoir in Odisha, India changed monthly in the year 2019.
display(dImage(data=open(out_gif,'rb').read(), format='png'))